Support collect and display the heartbeat thread status #17473

maobaolong · 2023-05-23T02:49:16Z

What changes are proposed in this pull request?

Support collect and display the heartbeat thread status

Why are the changes needed?

With this feature, we can insight the background heartbeat thread status and detect the issue before encountered an accident.

Does this PR introduce any user facing changes?

http://localhost:19999/api/v1/master/webui_heartbeat_threads
http://localhost:30000/api/v1/worker/webui_heartbeat_threads
http://localhost:20002/api/v1/job_master/webui_heartbeat_threads
http://localhost:30003/api/v1/job_worker/webui_heartbeat_threads

{
"debug": false,
"heartbeatThreadInfos": [
{
"count": 1,
"threadName": "Master Block Integrity Check",
"previousReport": "#0 [05-23-2023 09:59:13:190 - 05-23-2023 09:59:13:190 - 05-23-2023 09:59:13:209] ticked(s) 0, run(s) 0.",
"startTickTime": "05-23-2023 09:59:13:209",
"startHeartbeatTime": "",
"status": "WAITING"
},
{
"count": 3,
"threadName": "Master Cluster Metrics Updater",
"previousReport": "#2 [05-23-2023 10:00:13:164 - 05-23-2023 10:01:14:884 - 05-23-2023 10:01:19:769] ticked(s) 61, run(s) 4.",
"startTickTime": "05-23-2023 10:01:19:771",
"startHeartbeatTime": "",
"status": "WAITING"
},
{
"count": 1,
"threadName": "Master Log Config Report Scheduling",
"previousReport": "#0 [05-23-2023 09:59:13:255 - 05-23-2023 09:59:13:255 - 05-23-2023 09:59:13:255] ticked(s) 0, run(s) 0.",
"startTickTime": "05-23-2023 09:59:13:256",
"startHeartbeatTime": "",
"status": "WAITING"
},
{
"count": 1,
"threadName": "Master Lost Files Detection",
"previousReport": "#0 [05-23-2023 09:59:13:195 - 05-23-2023 09:59:13:195 - 05-23-2023 09:59:13:195] ticked(s) 0, run(s) 0.",
"startTickTime": "05-23-2023 09:59:13:195",
"startHeartbeatTime": "",
"status": "WAITING"
},
{
"count": 2,
"threadName": "Master Lost Master Detection",
"previousReport": "#1 [05-23-2023 09:59:13:251 - 05-23-2023 10:01:14:881 - 05-23-2023 10:01:19:715] ticked(s) 121, run(s) 4.",
"startTickTime": "05-23-2023 10:01:19:723",
"startHeartbeatTime": "",
"status": "WAITING"
},
{
"count": 3,
"threadName": "Master Lost Proxy Detection",
"previousReport": "#2 [05-23-2023 10:00:13:260 - 05-23-2023 10:01:14:882 - 05-23-2023 10:01:19:723] ticked(s) 61, run(s) 4.",
"startTickTime": "05-23-2023 10:01:19:723",
"startHeartbeatTime": "",
"status": "WAITING"
},
{
"count": 13,
"threadName": "Master Lost Worker Detection",
"previousReport": "#12 [05-23-2023 10:01:14:881 - 05-23-2023 10:01:19:713 - 05-23-2023 10:01:19:716] ticked(s) 4, run(s) 0.",
"startTickTime": "05-23-2023 10:01:19:729",
"startHeartbeatTime": "",
"status": "WAITING"
},
{
"count": 1,
"threadName": "Master Metrics Time Series",
"previousReport": "#0 [05-23-2023 09:59:13:214 - 05-23-2023 09:59:13:214 - 05-23-2023 09:59:13:232] ticked(s) 0, run(s) 0.",
"startTickTime": "05-23-2023 09:59:13:234",
"startHeartbeatTime": "",
"status": "WAITING"
},
{
"count": 110,
"threadName": "Master Persistence Checker",
"previousReport": "#109 [05-23-2023 10:01:25:462 - 05-23-2023 10:01:27:833 - 05-23-2023 10:01:27:836] ticked(s) 2, run(s) 0.",
"startTickTime": "05-23-2023 10:01:27:846",
"startHeartbeatTime": "",
"status": "WAITING"
},
{
"count": 110,
"threadName": "Master Persistence Scheduler",
"previousReport": "#109 [05-23-2023 10:01:27:830 - 05-23-2023 10:01:27:842 - 05-23-2023 10:01:27:846] ticked(s) 0, run(s) 0.",
"startTickTime": "05-23-2023 10:01:27:846",
"startHeartbeatTime": "",
"status": "WAITING"
},
{
"count": 3,
"threadName": "Master Replication Check",
"previousReport": "#2 [05-23-2023 10:00:13:212 - 05-23-2023 10:01:14:885 - 05-23-2023 10:01:14:885] ticked(s) 61, run(s) 0.",
"startTickTime": "05-23-2023 10:01:15:462",
"startHeartbeatTime": "",
"status": "WAITING"
},
{
"count": 1,
"threadName": "Master TTL Check",
"previousReport": "#0 [05-23-2023 09:59:13:192 - 05-23-2023 09:59:13:192 - 05-23-2023 09:59:13:193] ticked(s) 0, run(s) 0.",
"startTickTime": "05-23-2023 09:59:13:194",
"startHeartbeatTime": "",
"status": "WAITING"
},
{
"count": 0,
"threadName": "Master Throttle",
"previousReport": null,
"startTickTime": "05-23-2023 09:59:13:269",
"startHeartbeatTime": "05-23-2023 09:59:13:269",
"status": "STOPPED"
},
{
"count": 1,
"threadName": "Worker register stream session cleaner",
"previousReport": "#0 [05-23-2023 09:59:13:130 - 05-23-2023 09:59:13:130 - 05-23-2023 09:59:13:130] ticked(s) 0, run(s) 0.",
"startTickTime": "05-23-2023 09:59:13:134",
"startHeartbeatTime": "",
"status": "WAITING"
}
]
}

Base on this, we can create a web ui form for this heartbeat thread info.

ChunxuTang · 2023-05-23T20:38:39Z

Thanks for the work! Would you mind fixing the tests?

dbw9580 · 2023-06-09T03:55:22Z

pending review

jiacheliu3

Added a few comments, PTAL thanks!

core/common/src/main/java/alluxio/heartbeat/HeartbeatThread.java

jiacheliu3 · 2023-07-06T08:28:46Z

core/common/src/main/java/alluxio/heartbeat/HeartbeatThread.java

        mExecutor.heartbeat(limitTime);
+        long endHeartbeatTime = System.currentTimeMillis();


same as above

jiacheliu3 · 2023-07-06T08:33:37Z

core/common/src/main/java/alluxio/heartbeat/HeartbeatThread.java

+    mStartTickTime = 0L;
+    mStartHeartbeatTime = 0L;


why reset to 0 every time?

0 stand for that it is not start to heartbeat this period.

jiacheliu3 · 2023-07-06T08:40:56Z

core/common/src/main/java/alluxio/heartbeat/HeartbeatThreadManager.java

+    List<HeartbeatThread> list = HEARTBEAT_THREAD_INDEX_MAP.get(key);
+    if (list == null) {
+      list = new LinkedList<>();
+      HEARTBEAT_THREAD_INDEX_MAP.put(key, list);
+    }
+    HEARTBEAT_THREAD_MAP.put(heartbeatThread.getThreadName(), heartbeatThread);


map.computeIfAbsent(key, (k) -> new ArrayList<>());

jiacheliu3 · 2023-07-06T08:42:11Z

core/common/src/main/java/alluxio/heartbeat/HeartbeatThreadManager.java

+  private static final Map<String, HeartbeatThread> HEARTBEAT_THREAD_MAP
+      = new ConcurrentHashMap<>();
+  private static final Map<Object, List<HeartbeatThread>> HEARTBEAT_THREAD_INDEX_MAP
+      = new ConcurrentHashMap<>();


you use synchronized methods so no need for ConcurrentHashMap here?

jiacheliu3 · 2023-07-06T08:44:36Z

core/common/src/main/java/alluxio/wire/HeartbeatThreadInfo.java

+  }
+
+  @Override
+  public boolean equals(Object o) {


where do you need equals() and hashCode()?

Sorry, I am lazy copy the style from existing code, I just reference the file under alluxio/core/common/src/main/java/alluxio/wire, like AlluxioWorkerInfo BlockInfo and LogInfo

i think you can remove equals and hashCode

jiacheliu3 · 2023-07-06T08:52:34Z

core/common/src/main/java/alluxio/heartbeat/HeartbeatThread.java

        mExecutor.heartbeat(limitTime);
+        long endHeartbeatTime = System.currentTimeMillis();
+        mPreviousReport = String.format("#%d [%s - %s - %s] ticked(s) %d, run(s) %d.",


don't create report on run(), you may update the states but generate reports on demand

maobaolong · 2023-07-07T01:40:27Z

@jiacheliu3 Thanks for your review, I've addressed your comments, PTAL~

jiacheliu3

Thanks for the work! I added some comments for discussion.

jiacheliu3 · 2023-09-02T14:48:42Z

core/common/src/main/java/alluxio/heartbeat/HeartbeatThread.java

        mExecutor.heartbeat(limitTime);
+        mEndHeartbeatTime = CommonUtils.getCurrentMs();
+        updateState();


every tick you are resetting to zero? looks like a bug?

yeah, it looks need a structure to store the previous and current state

jiacheliu3 · 2023-09-02T14:52:38Z

core/common/src/main/java/alluxio/wire/HeartbeatThreadInfo.java

+  }
+
+  @Override
+  public boolean equals(Object o) {


i think you can remove equals and hashCode

jiacheliu3 · 2023-09-02T14:53:21Z

core/common/src/main/java/alluxio/wire/WebUIHeartbeatThreads.java

+public final class WebUIHeartbeatThreads implements Serializable {
+  private static final long serialVersionUID = -2903043308252679410L;
+
+  private boolean mDebug;


is this useful?

jiacheliu3 · 2023-09-02T14:55:05Z

core/server/master/src/main/java/alluxio/master/meta/AlluxioMasterRestServiceHandler.java

+    return RestUtils.call(() -> {
+      WebUIHeartbeatThreads response = new WebUIHeartbeatThreads();
+
+      response.setDebug(Configuration.getBoolean(PropertyKey.DEBUG));


will the output be different if you set PropertyKey.DEBUG? I dont think this is relevant?

jiacheliu3 · 2023-09-02T14:56:43Z

core/common/src/main/java/alluxio/heartbeat/HeartbeatThreadManager.java

+   */
+  public static Future<?> submit(ExecutorService executorService,
+      HeartbeatThread heartbeatThread) {
+    HeartbeatThreadManager.register(executorService, heartbeatThread);


i don't think it's a good idea to use a thread pool as key...

So, what can be the better key? Thread name?

jiacheliu3 · 2023-09-15T03:09:28Z

core/common/src/main/java/alluxio/heartbeat/HeartbeatThreadManager.java

+   * @return the heartbeat threads info
+   */
+  public static synchronized Map<String, HeartbeatThreadInfo> getHeartbeatThreads() {
+    SortedMap<String, HeartbeatThreadInfo> heartbeatThreads = new TreeMap<>();


double check if the thread is still alive, and make sure this ref is removed if the thread is already dead

maobaolong requested a review from jiacheliu3 May 23, 2023 02:50

maobaolong closed this May 24, 2023

maobaolong reopened this May 24, 2023

maobaolong force-pushed the heartbeatManager branch from 78f0c0d to fb2e430 Compare May 25, 2023 08:10

jiacheliu3 requested a review from dbw9580 May 26, 2023 03:24

maobaolong force-pushed the heartbeatManager branch from c2929a1 to 4d6c067 Compare May 26, 2023 07:12

Support display the heartbeat thread status

499db76

maobaolong force-pushed the heartbeatManager branch from 4d6c067 to 499db76 Compare June 5, 2023 01:12

Fix test

2110f05

jiacheliu3 requested changes Jul 6, 2023

View reviewed changes

Address comments

e5adfc0

maobaolong requested a review from jiacheliu3 July 7, 2023 01:40

maobaolong added 2 commits July 14, 2023 11:04

Address comments

2454be7

Merge remote-tracking branch 'upstream/master-2.x' into heartbeatManager

4deec80

jiacheliu3 requested changes Sep 2, 2023

View reviewed changes

Address comments

4b2a3dc

jiacheliu3 reviewed Sep 15, 2023

View reviewed changes

Merge remote-tracking branch 'upstream/master-2.x' into heartbeatManager

f0f03b3

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support collect and display the heartbeat thread status #17473

Support collect and display the heartbeat thread status #17473

maobaolong commented May 23, 2023 •

edited

Loading

ChunxuTang commented May 23, 2023

dbw9580 commented Jun 9, 2023

jiacheliu3 left a comment

jiacheliu3 Jul 6, 2023

maobaolong Jul 7, 2023

jiacheliu3 Jul 6, 2023

maobaolong Jul 7, 2023

jiacheliu3 Jul 6, 2023

maobaolong Jul 7, 2023

jiacheliu3 Jul 6, 2023

maobaolong Jul 7, 2023

jiacheliu3 Jul 6, 2023

maobaolong Jul 7, 2023

jiacheliu3 Sep 2, 2023

jiacheliu3 Jul 6, 2023

maobaolong commented Jul 7, 2023

jiacheliu3 left a comment

jiacheliu3 Sep 2, 2023

maobaolong Sep 4, 2023

jiacheliu3 Sep 2, 2023

jiacheliu3 Sep 2, 2023

maobaolong Sep 4, 2023

jiacheliu3 Sep 2, 2023

jiacheliu3 Sep 2, 2023

maobaolong Sep 4, 2023

jiacheliu3 Sep 15, 2023

		mExecutor.heartbeat(limitTime);
		long endHeartbeatTime = System.currentTimeMillis();

Support collect and display the heartbeat thread status #17473

Are you sure you want to change the base?

Support collect and display the heartbeat thread status #17473

Conversation

maobaolong commented May 23, 2023 • edited Loading

What changes are proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user facing changes?

ChunxuTang commented May 23, 2023

dbw9580 commented Jun 9, 2023

jiacheliu3 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

maobaolong commented Jul 7, 2023

jiacheliu3 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

maobaolong commented May 23, 2023 •

edited

Loading